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ABSTRACT 

We  propose  to  incorporate  a  weighted  difference  of  anisotropic 
and  isotropic  total  variation  (TV)  norms  into  a  relaxed  formu¬ 
lation  of  the  two  phase  Mumford-Shah  (MS)  model  for  image 
segmentation.  We  show  results  exceeding  those  obtained  by 
the  MS  model  when  using  the  standard  TV  norm  to  regular¬ 
ize  partition  boundaries.  In  particular,  examples  illustrating 
the  qualitative  differences  between  the  proposed  model  and 
the  standard  MS  one  are  shown.  A  fast  numerical  method 
is  introduced  to  minimize  the  proposed  model  utilizing  the 
difference-of-convex  algorithm  (DCA)  and  the  primal  dual 
hybrid  gradient  (PDHG)  method. 

Index  Terms —  Total- Variation,  Segmentation,  Mumford- 
Shah  Functional,  Chan-Vese  Model,  Primal-Dual 

1  Introduction 

The  celebrated  Mumford  and  Shah  (MS)  model  [1]  in  2-phase 
form  known  as  the  Chan-Vese  (CV)  model  [2]  is  one  of  the 
most  studied  and  successful  models  in  image  processing.  The 
CV  model  has  the  following  formulation: 

min  Per(E)  +  A  [  (ci  —  f)2dx  +  f  (c2  —  f)2dx.  (1) 

e>ci>c2  7e  7ec 

Here,  /  is  a  given  gray  scale  image,  E  denotes  a  region,  and 
Ec  the  outside  of  that  region.  The  key  idea  is  to  minimize 
the  above  functional  (1)  by  matching  two  regions  (constants) 
in  the  L 2  sense  while  also  minimizing  the  perimeter  of  the 
boundaries  between  them.  The  values  are  designated  by  c\ 
and  C2  and  are  obtained  on  the  regions  E  and  Ec  respectively. 
The  two  regions  and  values  are  unknowns  and  need  to  be 
solved  for  during  the  minimization.  In  general,  the  CV  model 
is  difficult  to  implement  in  practice.  Chan  and  Vese  utilized 
a  level  set  method  formulation  and  showed  successful  results. 
Conventionally,  level  set  methods  can  be  slow  to  converge  due 
to  the  need  for  periodic  reinitialization  of  the  level  set  func¬ 
tion  to  a  signed  distance  one.  More  recently,  Chan  et  al.  [3] 
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and  Bresson  et  al.  [4]  proposed  convex  relaxed  formulations 
of  the  CV  model.  The  general  formulation  follows  as: 

min  [  g\\7u\dx-\-  X  [  {(ci  —  f)2  —  (c2  —  f)2}  u  dx 
0<u<l Jn  JQ 

(2) 

where  ft  is  the  image  domain  and  g{pc)  is  an  edge  detector 
or  set  to  value  1.  The  segmented  region  E  is  realized  by 
taking  the  upper  level  set  of  u  in  the  following  way:  E  = 
{x  :  u{x)  >  1/2}.  The  main  idea  was  to  relax  the  non-convex 
constraint  of  u  being  binary  to  a  convex  one.  Once  the  min¬ 
imization  problem  (2)  is  solved,  one  simply  thresholds  u  to 
obtain  the  segmented  region.  The  authors  show  accurate  and 
successful  segmentations  along  with  fast  numerical  results. 

The  L0  norm  on  the  gradient  J(u)  =  ||  Vu\\o  is  construed 
as  the  length  of  partition  boundaries  in  the  context  of  the  clas¬ 
sical  Potts  model  [5]  or  two-phase  Mumford-Shah  segmen- 
taion  [1].  To  circumvent  the  NP-hard  Lq  norm,  the  convex  re¬ 
laxed  approach  is  to  use  the  L\  norm  on  Vu,  see  [6].  L\  norm 
on  the  gradient  is  known  as  the  total- variation  (TV)  norm  [7] . 
To  approximate  Lq,  a  series  of  works  show  that  L\  —  L2  is 
better  than  Lo  (greedy  approaches),  L\,  and  Lp  in  compres¬ 
sive  sensing  [8].  When  applying  L\  —  L2  on  image  gradients, 
it  enforces  gradients  to  be  either  horizontal  or  vertical.  To 
account  for  other  gradient  directions,  a  weighted  difference, 
i.e.  Li  —  aL2  is  considered  in  the  recent  work  by  Lou  et 
al.  [9],  where  a  is  chosen  according  to  gradient  distributions. 
They  show  accurate  results  and  provide  a  convergence  proof 
using  a  difference-of-convex  algorithm  (DCA)  [10,  11,  12]. 
L\  —  O.5L2  is  also  observed  to  be  numerically  more  stable 
compared  to  Li  —  L2.  We  also  note  that  the  level  curves  for 
the  convex  addition  L\  +  olL2  closely  resemble  those  of  Li 
while  those  for  L\  —  aL2  are  closer  to  L0. 

This  work  is  an  extension  of  the  work  by  Lou  et  al.  [9] 
to  the  image  segmentation  problem.  To  that  end,  we  propose 
incorporating  a  weighted  difference  of  anisotropic-isotropic 
TV  into  the  two-phase  relaxed  CV  model.  For  the  remainder 
of  the  paper,  we  refer  to  L\  —  aL2  as  the  weighted  difference 
of  anisotropic  and  isotropic  TV: 

J ani  CiJiso  —  \\ux  ||  1  T~  || Uy  ||  1  <t||  'sj \ux  \  2  T~  \^y  \  2  ||  1  (2) 


where  it  is  understood  that  these  norms  are  operating  on  the 
gradients  of  the  image.  Here,  a  G  [0, 1]  and  is  chosen  based 
on  gradient  distributions. 

Thus,  the  key  contributions  of  this  paper  are: 

•  Introduce  a  variant  of  the  relaxed  CV  model  that  geo¬ 
metrically  better  preserves  boundaries  of  regions  over 
the  standard  CV  model. 

•  Presentation  of  new  numerical  schemes  utilizing  a 
DCA  algorithm  with  primal  dual  hybrid  gradient 
(PDHG)  methods  to  solve  sub-problems  efficiently. 
DCA  has  provable  convergence  properties  in  certain 
settings. 

•  The  method  can  be  executed  with  complete  automation. 
Related  work  includes  the  preceding  CV  model  [2]  and 

convex  relaxed  variants  [3,  4].  A  recent  hybrid  ADMM  and 
dynamic  programming  to  solve  the  Potts  model  is  introduced 
by  Storath  et  al.  in  [13]  while  convex  relaxed  approaches  are 
found  in  Pock  et  al.  [14]  and  Chambolle  et  al.  [15]. 

The  paper  is  organized  as  follows,  in  section  2  we  intro¬ 
duce  the  proposed  model  along  with  the  methodology  to  min¬ 
imize  it.  Numerical  results  are  shown  in  section  3.  Lastly, 
conclusions  and  future  work  can  be  found  in  section  4. 

2  Relaxed  Anisotropic-Isotropic 
Chan-Vese  Model  (AICV) 

We  propose  a  variant  of  the  relaxed  CV  model  proposed  by 
Chan  et  al.  [3]  and  the  CVG  model  proposed  by  Bresson 
et  al.  [4].  Given  an  initial  image  /  defined  on  domain  Q 
containing  a  region  to  be  segmented  that  we  denote  by  E,  the 
AICV  model  seeks  to  minimize  the  following: 

min  /  \ux(x)\  +  \uy(x)\  —  a\\7u(x)\ dx 
0<u<lJQ 

+  A  [  {(ci  -  f{x))2  -  (c2  -  f(x))2}  u(x)  dx 
r1(x,c1,c2) 

=  min  F(u,c1,c2,\).  (4) 

0<u<l 

The  anisotropic-isotropic  (Al)  term  fQ  \ux\  +  \uy\  —  a\\7u\ 
denotes  a  regularization  of  the  region  E  and  enforces  regu¬ 
larity  on  the  boundary  that  separates  the  regions  where  the 
two  values  c\  and  c2  are  taken.  We  outline  the  minimization 
method  and  subsequent  algorithm  below. 

Assuming  that  either  ci,  c2  and  A  are  known  or  c\  and 
c2  will  be  solved  for  during  the  minimization,  we  refer  to  the 
objective  functional  F(u,  ci,  c2,  A)  as  F(u)  and  ri(x,  ci,  c2) 
as  ri(x).  Then  the  above  AICV  model  (4)  can  be  written 
as  a  minimization  of  a  difference-of-convex  (DC)  functionals 
F(u)  =  G{u )  —  H(u)  in  the  following  manner: 

f  G(u)  =  IKItjH-  IKHi  +c\\u\\l  +  X(r1,  u) 

\  H(u)=a\\\7u\\2tl+c\\u\\2  (  ) 

where  c  is  a  small  parameter  in  front  of  H^Hl  enforcing  strong 
convexity  on  G  and  H  in  the  difference.  We  linearize  the 


strongly  convex  term  in  H (u)  and  obtain  the  following  itera¬ 
tive  scheme: 

Un+1  =  argmin  ||ux||i  +  ||uji  +  c\\u\\%  +  A(rx,  u) 

o<«<!  (6) 

-a||Vu||2,i  -2c(u,  un). 

The  aforementioned  minimization  associated  to  the  iterative 
scheme  in  model  (6)  has  the  following  equivalent  uncon¬ 
strained  split  formulation: 

Un+1  =  argmin  11^ 111  +  ||ny  Jj  +  c\\u\\2  +  A(n, v) 

U,V 


+  h\\u~  v\\l  +^(I'0)>1)  -a||Vu||2li  -  2 c(u,un).  (7) 
Here  z/(£)  =  max{0,  2|^  —  ||  —  1}  is  an  exact  penalty  given 
that  /?  is  chosen  large  enough,  see  [4] .  Each  DCA  subproblem 
in  (7)  can  be  solved  by  solving  the  two  subproblems: 

1.  v  being  fixed,  we  search  for  u  as  a  solution  of: 

minll^Hi  +  |K||i  -  a||V«||2,i  +  ^\\u  -  vf2 

+  c\\u\\l  -  2 c(u,  un),  (8) 

2.  u  being  fixed,  we  search  for  rasa  solution  of: 

min  A  || u  -  ©||2  +  A(n,  v)  +  /3  [  v(v)dx.  (9) 
v  2  0  Jn 

The  solution  to  (9)  is  given  by  a  simple  shrinkage  scheme: 

v  =  min  |  max  ^u(x)  —  6\ri(x,  ci,  c2),  o|,  l|.  (10) 

The  solution  to  (8)  is  obtained  by  a  primal-dual  hybrid  gradi¬ 
ent  (PDHG)  method  by  Zhu  and  Chan  [16]  which  we  focus 
on  now.  More  on  the  PDHG  method  and  variants  thereof  can 
be  found  in  [17]. 

The  minimization  problem  in  (8)  reduces  to  the  following 
min-max  problem: 

min  max  (u,  — di vp  -\-adivq)-\ — —  ||ix  —  ^||i 

u  pGX,gGX  2# 

+  c\\u\\l-2c{u,un)  (n) 

=  min  max  $>(u,p,q), 

u  pGX,  gGX 


where  X  =  {p  =  (pi,P2)  •  \pi  \  <  1  &  \P2\  <  1}  and  X  = 
{q  =  (^1,^2)  :  \q  \  <  1}-  We  will  optimize  (11)  in  the  fol¬ 
lowing  two  step  manner  below. 

Step  1.  Dual  Step 

Part  A.  Fix  u  —  uk  and  q  =  qk  and  apply  one  step 
of  projected  gradient  ascent  to  max  problem: 

max  <&(uk,p ,  q  k).  (12) 

PGX 

The  projected  gradient  ascent  for  the  maximization 
(12)  is  simply: 

pk+1  =P^(pk  +rkVuk)  (13) 


where  is  a  time  step  and  the  projection  is  obtained 
by  the  following  operation: 


= 


pi 


P2 


(14) 


^maxflpi),  1}  ’  max{|p2|,l}, 

Part  B.  Fix  u  =  uk  and  p  =  pk+1  and  apply  one 


step  of  projected  gradient  ascent  to  max  problem: 

ma -k  <&(uk ,pk+1  ,q).  (15) 

<?£X 

The  projected  gradient  ascent  for  the  maximization 
(15)  is  simply: 

<ffc+1  =  PxOffe  +  Tk(-aVuk))  (16) 


where  t\~  is  a  time  step  and  the  projection  is  obtained 
by  the  following  operation: 


Px(q)  = 


maxjlla  ||, !}' 


(17) 


Step  2.  Primal  Step 

Fix  p  =  p fc+1  and  q  =  q  fc+1  and  apply  one  step  of  gradi¬ 
ent  descent  method  to  the  minimization  problem: 


min  p 


k+ 1  -*/c+ 1 


)• 


(18) 


The  gradient  descent  associated  to  the  minimization  (18) 
is  given  by: 

uk+ 1  =  (1  —  6»fc  —  2  cOkO)uk 

+  9k  (v  —  6{— divj?fe+1  +  adiv(ffe+1  —  2cun)^ 

where  Ok  is  a  time  step.  The  PDHG  algorithm  for  min¬ 
imizing  the  unconstrained  AICV  (7)  model  is  shown  in 
Algorithm  1.  We  note  that  when  choosing  time  steps  Tk 
and  Ok,  one  can  utilize  dynamic  time  stepping  to  acceler¬ 
ate  convergence.  Guidelines  are  given  in  [17]. 


Algorithm  1  DCA  to  minimize  unconstrained  AICV  (7) 

Initialization:  Pick  u°,  MaxDCA,  MaxPDHG,  step  size  Tk 
and  Ok,  and  set  u  =  0. 
for  1  to  MaxDCA  do 

Set  v°  =  p 0  =  q°  =  0  and  k  <—  0 

for  1  to  MaxPDHG  do 

vk+1  =  min  {  max  { uk  —  0\r\(x ,  ci,  C2),  0},  l} 
Pk+1  =  Pyi{pk  +  rkTuk) 
qk+1  =  Px(qk  +  Tk(-aS/uk )) 
uk+ 1  =  (1  —  6»fe  —  2  c0k0)uk 

+  9k(vk+1  —  9(—div  pk+1  +a:div<ffc+1  —  2cm)) 

end  for 

u  =  uk+1 

end  for 


3  Experiments 

In  this  first  example  we  segment  a  synthetic  image  to  show 
some  properties  associated  to  the  proposed  model.  Observed 
in  Fig.l  (a)  is  a  clean  square  image  while  a  medium  noisy 
version  with  initial  contour  is  in  (b);  Gaussian  noise  a  =  0.5. 
In  Fig.  1  (c)  the  segmentation  from  the  standard  CV  model  is 
observed  where  straight  boundaries  are  captured  but  corners 
are  rounded  off.  What  is  well  known  about  the  TV  norm  in 
regard  to  denoising  also  manifests  itself  in  the  segmentation 


(a)  clean  image  / 


(b)  noisy  /  &  initial  contour 


(c)  TV  (cr  =  0.5) 


(d)  Li,Li  —  aL2  (cr  =  0.5) 


(e)  TV  (a  =  0.5) 


(f)  Li,Li  -aL2  (o  =  0.5) 


(g)  TV  (a  =  0.65) 


(h)  L i  -  0.5L2  (cr  =  0.65) 


Fig.  1:  Square  Segmentation  Example. 


problem,  see  [18].  The  segmentation  results  for  the  L\  and 
Li  —  aL2  (a  =  1,0.5)  CV  models  are  seen  in  Fig.  1  (d) 
where  perfect  segmentations  are  observed.  Here,  the  error 
1 1 /clean  —  {zz  >  1/2}  ||2  =  0  while  that  for  the  TV  norm  model 
is  larger  than  zero.  In  Fig  1  (e)  and  (f)  zoom-ins  of  the  top 
left  corner  of  (c)  and  (d)  respectively  where  TV  in  (c)  cannot 
capture  the  corner.  No  amount  of  tuning  A  in  this  case  will 
yield  a  perfect  segmentation.  In  Fig  1  (g)  and  (h)  we  segment 
the  same  image  under  heavier  noise,  cr  =  0.65.  In  (g)  the  TV 
norm  model  prominently  cuts  off  the  corners  while  in  (h)  the 
proposed  model  captures  them  with  much  higher  accuracy. 

We  now  look  at  a  real  example  and  segment  a  video  frame 
obtained  from  a  sequence  under  atmospheric  turbulence.  In 
Fig.  2  (a)  the  frame  is  observed  and  in  (b)  a  zoomed  and  pro¬ 
cessed  version  from  the  method  in  [19]  is  seen  with  initial 


(a)  clean  image  / 


(b)  noisy  /  &  initial  contour 


Fig.  2:  Sign  Segmentation  Example. 


Fig.  3:  Sign  Segmentation  Example  Zoom-Ins. 


contour.  In  (c),  (d),  (e),  and  (f)  are  the  segmentation  results 
obtained  from  the  CV  model  with  TV  norm,  L\,  L\  —  L2,  and 
L\—  0.51/2  norms  respectively.  All  methods  capture  the  large 
bars,  some  numbers,  and  smaller  squares.  Upon  closer  in¬ 
spection,  differences  are  observed  like  in  Fig.  3  (a)-(d)  where 
zoom-ins  of  the  segmentations  from  the  CV  model  with  TV 
norm,  Li,  L\  —  L2,  and  L\  —  0.51^2  norms  respectively  are 
seen.  For  the  top  3  squares  in  (a),  the  TV  norm  does  not  cap¬ 
ture  the  geometry  completely.  The  L\  —  L2  in  (c)  does  a  better 
job  than  TV  while  L\  in  (b)  and  L\  —  0.51/2  in  (d)  capture  the 
squares  most  accurately.  The  largest  contour  in  (b)  from  the 
L\  norm  appears  blocky  with  the  lower  contour  cutting  off 
horizontally  versus  the  smoother  version  found  in  (d)  from 
the  Li  —  0.51/2  norm.  In  Fig.  3  (e)  and  (f)  are  zoom-ins  of 
the  results  from  TV  and  L\  —  0.51/2  on  one  large  vertical  bar 
where  TV  cuts  off  the  corner  and  L\  —  0.5 L2  has  straighter 
geometry  keeping  the  left  corner.  We  deduce  that  with  TV, 
some  geometry  is  lost  but  it  affords  smooth  boundaries  while 
Li  prefers  blocky  geometry  but  doesn’t  favor  smooth  tran¬ 
sitions.  L\  —  I/2  mimics  TV  with  some  blocky  geometry 
preservation.  L\  —  0.5L2  has  attributes  of  L\  in  that  it  can 
preserve  blocky  geometry  yet  simultaneously  keep  smoother 
transitions;  a  best  of  both  worlds  if  you  will. 

In  all  experiments,  A  was  tuned  to  keep  as  much  geometry 


and  features  in  the  segmentation.  5000  iterations  were  used 
for  TV  and  Lu  while  MaxDCA  =  5  and  maxPDHG  =  1000 
for  Li  —  L2  and  L%  —  0.5 L2  with  fixed  PDHG  time-steps 
Tk,  0k  =  1/8.  We  fixed  the  splitting  parameter  6  =  0.1  in 
all  experiments.  Constants  c\  and  C2  were  fixed  with  c\  =  1 
and  C2  =  0  for  the  square  example  while  they  were  c\  = 
8.5  x  10-2  and  C2  =  7.15  x  10-1  for  the  sign  experiment. 

4  Conclusions  and  Future  Work 

We  proposed  a  relaxed  version  of  the  CV  model  utilizing 
a  difference  of  convex  anisotropic  and  isotropic  TV  norms. 
Moreover,  we  showed  compelling  examples  illustrating  qual¬ 
itative  differences  between  the  proposed  model  and  the  stan¬ 
dard  CV  one.  A  fast  algorithm  by  way  of  DCA  and  PDHG 
was  also  introduced  for  its  minimization.  We  note  that  one  of 
the  inner  minimizations  (8)  is  non-convex  but  seems  to  con¬ 
verge  to  a  minimum  consistently  in  practice.  Future  work 
includes  a  contrast  between  a  weighted  difference  of  convex 
L\—  olL2  TV  norms  and  a  weighted  sum  olL\  +  (1  —  a)L2  of 
TV  norms  i.e.  a  fQ(\ux\  +  \uy\)dx+(l—a)  \S7u\dx.  Qual¬ 
itatively,  the  results  are  comparable  but  the  level  curves  for  the 
difference  more  closely  resemble  those  of  the  Lq  norm.  The 
advantage  of  the  sum  is  convexity  and  this  detailed  compari¬ 
son  will  be  found  elsewhere.  We  also  plan  on  proving  conver- 


gence  using  the  DCA  framework  for  outer  iterations  on  the 
difference  of  convex  TV  norms  model.  For  solving  the  sub¬ 
problems,  we  will  be  using  Bregman  [20,  21]  inner  iterations 
where  the  non-convex  term  is  linearized  yielding  a  convex 
subproblem.  A  threshold  result  like  in  [3]  along  with  exten¬ 
sions  of  the  proposed  model  to  the  piecewise  smooth  MS  and 
multiphase  Potts  models  are  also  in  the  works. 
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